Lane line detection at nighttime on fractional differential and central line point searching with Fragi and Hessian

To detect lanes at night, each detecting image is the fusion of the multiple images in a video sequence. The valid lane line detection region is identified on region merging. Then, the image preprocessing algorithm based on the Fragi algorithm and Hessian matrix is applied to enhance lanes; to extract the lane line center feature points, the image segmentation algorithm based on Fractional differential is proposed; and according to the possible lane line positions, the algorithm detects the centerline points in four directions. Subsequently, the candidate points are determined, and the recursive Hough transformation is applied to obtain the possible lane lines. Finally, to obtain the final lane lines, we assume that one lane line should have an angle between 25 and 65 degrees, while the other should have an angle between 115 and 155 degrees, if the detected line is not in the regions, the Hough line detection will be continued by increasing the threshold value until the two lane lines are got. By testing more than 500 images and comparing deep learning methods and image segmentation algorithms, the lane detection accuracy by the new algorithm is up to 70%.


Lane line image features and multiple image fusion
At night, the contrast of the area of interest will be much lower than that at daytime, within the scope of the street lamp and the place near the lamp, the visual information will be richer, while the information on both sides, especially when it is far away from the lamps, will be weaker. The features of the lane line image at night are as follows: (1) It is different from the daytime, at night, the lane lines in a road image has weak information. Due to the influence of lights, the white color lane lines are often in gray color. Compared with other parts of the road surface, the color and reflectance of the lane lines are much weaker than that at daytime, that is to say, the difference between the pixel values of the lane lines and that of the road surface on both sides is relatively small. (2) Due to the influence of building shadows, tree shadows, wear of lane marking line itself, strong lights, brake trace lines, fog, haze, sandstorm and rain, etc., the road traffic noise and interference in a lane line image are very strong. (3) In the area close to the vehicle, sometimes, referred to as the nearsighted area, the road information is fuzzy compared with that at daytime, which affects the accuracy of lane line detection; while, in the area far away from the vehicle, due to the strong reflection of street lights, this part of the image reflects the road information well. (4) According to the test of different highway, there are different standards for lane length, width and lane segment spacing, as shown in Table 1. Even if the lane line is not worn, it is difficult to ensure that every video image has the two lane lines, but the probability of acquiring lane lines also depends on the speed of vehicle. When the video is PAL system, 25 images/second can be collected, and when the video is NTSC system, 30 images/second can be acquired. According to Table 2 Generally, as the vehicle speed increases, the lane line view will e worse, and the lane line detection will be harder. We take the middle vehicle speed 80 km/h as an example, and the length of a white solid line is 4 m plus the interval distance of 6 m, total 10 m. Eleven images can be obtained continuously by a PAL system to cover the whole 10 m length in one second, so an image can be selected about every three images and the three images can be used to form the detecting image. During this period, the vehicle moving distance is about 10 m, which is more than twice of the length of a car, and more than 1/3 of that of a 24 m trailer. During this time interval, the vehicle cannot deviate from the lane line too much and it is difficult to run out of the lane lines too much. In Fig. 1, there are four blurred road images at night, the length of the lane line in each image is very short, or there is no lane line on one side or both sides, which is difficult or even impossible for the lane line detection. The interval of each three of images is within ten images, therefore, after merging the three images, the result in Fig. 2 can be obtained, so the merging result image is better.  1  600  10-20  900  75  10-15  15   2  400  10-20  600  75  10-15  15   3  300  10-20  400  75  10-15  15   4,5  100-200  10-20  300  75 10- 15 15

Image smoothing and enhancement based on Fragi and Hessian matrix
As Fig. 3 shown, the images at night are much different to that at daytime, the images are dark, and the lane line signals are weaker. From the both 2D image and 3D graph, we can see that the lines at daytime are very clear, their gray scale values are much higher than that in background in the 2D image, and the lane line depths at daytime are much deeper than that in the other regions in the 3D graph. In Fig. 3b, when the lane lines are very short, the signals even cannot be seen in the 3D graph. Therefore, we have to do enhancement for the lane lines in the image at night. Because the contrast of the image at night is poor and the noise is much. Before extracting the lane lines, the image preprocessing should be carried out. The algorithm should includes three basic procedures: image smooth for denoising, contrast stretch and lane line enhancement, to do these, the following Fragi and Hessian matrix based algorithm will be adopted. The specific procedure is as follows 33,34 .
For an array I : → R, ∈ R 2 , a Gaussian kernel g(p; σ ) = 1 √ 2πσ e − x 2 +y 2 2σ 2 is applied firstly, and then the Hessian matrix corresponding to I at point p(x, y) is set as: The definition of lane lines is as follows: where, t d (p, θ; σ ) = g xx cos 2 θ + g yy sin 2 θ + g xy sin 2θ.
The forward filter t f (p; σ , ψ 1 ) and the backward filter t b (p; σ , ψ 2 ) are as: where, ψ 1 , ψ 2 are angel adopted for detecting the evidence of lane lines in the neighbor pixels. d is an offset parameter which is set at a appropriate value. It is difficult to contribute enough information when d value is low. It can cause the incorrect segmentation for judging the spurious lane line pixels into the real lane lines.
As the response, the two oriented filters are given by T f (p; σ , ψ 1 ) = t f (p; σ , ψ 1 ) * I(p) and T b (p; σ , ψ 2 ) = t b (p; σ , ψ 2 ) * I(p) respectively, and the enhanced lane lines are in Eq. (5) : (1) H σ (p) = g xx (p) g xy (p) g xy (p) g yy (p) * I(p)  www.nature.com/scientificreports/ Then we search for the max response at the multi-orientations as the output of image. In Fig. 4, the typical lane line image enhancement procedure is presented, the original image quality is bad, but their preprocessing results, such as Histogram transformation, Fragi enhancement and noise removal, are satisfactory.

Search of lane line feature points on fractional differential
Generally, since the gray scales are not uniform 35 in the lane lines, the more the lane line points we search in a certain area, the lines can be more easily identified. In order to collect as many lane line feature points as possible, we study an algorithm: firstly, the image is inverted to make the lane lines as low gray scales, and then the lane line feature points are detected based on Fractional differential. The specific detection routine is as follows.
In Fig. 5a, it is a 9 × 9 template, in which there are four square areas of different sizes around the center pixel, and the template should be large enough to detect whether the center pixel is a valley edge candidate. For valley edge detection, 81 (9 × 9 template) pixels for calculation may be too large, even if a lot of information is used, but the valley edge detection results are not satisfactory. On the contrary, based on (a), we also test a 7 × 7 square template shown in Fig. 5b, it still needs many pixels (49) for the calculation. Instead of that, we can apply a circular template for the detection, which is more suitable for the actual situation, and can utilize fewer pixels than that in a same sized square area. There are three circular areas (3 × 3, 5 × 5 and 7 × 7) around the central pixel. Since the valley edge point has its four different directions, and the four directions are marked in (b). As an example in Fig. 5c, we mark two trapezoidal areas based on (b), which can be used for valley edge detection in the vertical direction (AB in (b)), because we mark "1", "2" and "3" lines in the top trapezoidal area (red color) and the bottom area (blue color ). If the detection pixel "0" is the lowest point, its gray scale should be lower than that in "1" lines. The gray scales in "2" lines should be lower than that in "3" lines. The remaining question is how to calculate the weighted average gray scale value of each line. An example of 5 × 5 templates is given as the follows.
Suppose there is a lane line center point P in the vertical direction in Fig. 5c, we have three detection lines in Fig. 6, they are ab, cd and ef corresponding to lines "1", "2" and "3" respectively in Fig. 6. In the trapezoid area at the top (Fig. 5c), we have orthogonal lines aP, cP and eP in Fig. 6, which meets the conditions of aP < cP < eP in Fig. 6, otherwise P is not the center point of the lane line, of course, if the condition bP < dP < fP is not met, it is not enough to determine that P is the centerline feature point of the lane line. To determine if point P is the centerline feature point of the lane line, we study the following method.
For the gray scale value of each line in Fig. 5c and Fig. 6, it should be a weighted averaging gray scale value, the weight of the central pixel should be larger, and the remaining pixel values should be smaller. Since a lot of literature report that the Fractional differential calculus is good for smoothing thin edges, hence we calculate the coefficients based on Fractional differential. In this study, we use Grümwald-Letnikov (G-L) definition 36,37 as the following. www.nature.com/scientificreports/ can meet the condition (m + 1) < m ∈ Z , Z represents for the continuous derivative of the integer set order; if v > 0 and m is equal to [v] , then v order derivative is: , the signal duration [a, t] is divided equally in h = 1 , the unit equal interval: Hence, v order fractional order of the differential expression in 1D signal s(t) is deducted as: The n + 1 non-zero coefficient values can be in order as: We make the absolute values: a 0 = 1,a 1 = |−v|,a 2 = v 2 − v /2 , when v = 0.5 , we obtain a 1 = 0.5 , a 2 = 0.125 , in order to remove the decimals, for line "1", we enlarge all digits for 2 times, then we got b 0 = 2a 0 = 2 ,b 1 = 2v = 1 ; and for line "2", we enlarge all digits for 8 times, then we obtain  www.nature.com/scientificreports/ The valley or ridge detection algorithms have been used in different applications, but for this application, we study a special algorithm, which is different to others 38,39 , as described as the follows.
As Fig. 7 shown, the templates for four directions are illustrated. Where, we define detecting point or central pixel as x 0 , line "1" as x 1 and line "2" as x 2 , in the vertical direction (Fig. 7b), the top part (Fig. 5c, or red color trapezoid region) is taken as an example for the valley edge point detection. f i, j is a gray scale image for input, and g i, j is the binary image for output.
In the vertical direction, we have two values (at the top region and the bottom region, see Fig. 5c), we call them as y +90 and y −90 , if y +90 > 0 and y −90 > 0 , we have y 90 = y +90 + y −90 . In the same way, we calculate the other three directional y values. Then, we compute: To output a gradient magnitude image, we do: If we output a binary image directly, when we set a threshold T, we can do: It is normal that an original image include a lot of noise which will affect valley edge detection result. One simple way for reducing the noise is to use a smoothing filter such as the Gaussian smoothing function, which has a width parameter sigma, often referred to as the scale space parameter. The choice of sigma depends on white spot size distribution. Figure 8 gives the comparison between the new algorithm and other traditional algorithms for the two lane lines in the image at night.
In Fig. 8a, there are two vague lane lines in the original image (the vehicle speed is 60-80 km/h). Otsu thresholding segmentation result shows that the image illumination is uneven, the gray scale value of the bottom left corner is lower, and the gray scale value of the middle part is shallow, as shown in Fig. 8b. In Fig. 8c, the image is a binary image obtained by the Canny edge detector. For the Canny operation, we give a lower threshold value, and the double edges of most areas of lane lines are displayed 40 . However, there is too much noise in the result image, which is difficult to remove by the post-processing functions. In Fig. 8d, the Minimum Spanning Tree (MST) algorithm (Graph based algorithm) 41 is applied to segment the image. In the result image, some segments of the two lane lines are detected (the green segment on the left and the purple segment on the right), but the

Experiments and analysis
As above description, the lane line detection at night is harder, hence we study a new method for the lane line extraction, and the method working procedure is presented in Fig. 9. The method mainly includes two parts as the dashed line rectangles shown, the first part is to extract valuable detecting regions, which can reduce the method calculation burden and remove noise which can greatly affect the detection results. In Fig. 9,   The third and fourth rows in Fig. 10 are Canny detection results with different high and low thresholds 40 . It can be seen that the image segmentation algorithms based on discontinuity are superior to the algorithms based on similarity. When Canny's thresholds are low, although the edges of the targets can be detected, but there are many noise edges in the images, that is over segmentation or over detection, which is hard or even impossible for the post-processing. However, when the thresholds are selected as higher values, although the noise edges are much less, the boundaries of some targets are missed, that is under segmentation or missing detection. So, it is not enough to provide complete information for lane line identification.
The fifth row is for the results of FCM (fuzzy clustering) 42 . It has the similar effect with the global thresholding results, the regions with high gray scales in the upper part and the middle part are detected as targets, and the lane line (or part of the segment) is not fully extracted. Although the algorithm is effective for complex multi-target image segmentation, it cannot achieve the desired effect for the situation of slender target and low contrast images. The sixth row is for the detection result images of clustering analysis 42 . Different from FCM, it can segment multi-targets as many as possible, so it can extract some lane lines (or some line segments), but the lane line extraction is not completed, or lane lines are fused into other targets, only some points (spots) and a part of lines (line segments) are detected. Hence, although its detection results are better than that by the above-mentioned FCM algorithm, but for this kind of special images, we should have some improvements for the algorithm to get better results.
In the seventh row, the results are obtained by MST (Minimum Spanning Tree) algorithm 41 . Compared with the above Clustering analysis algorithm, the effect of image segmentation is improved, but the extraction of lane lines is not completed. Even if the Hough transform is used to detect lane lines in the post-processing, it is difficult to achieve the required effect due to the impact of noise targets, so there is still a lot of room for improvement.
The eighth row presents the results based on the detection of the characteristic points on the lane line by the algorithm studied in this paper. Its main idea is to detect as many feature points as possible on the lane lines, to find as many points on the canyon line or the center of the potholes in the canyon as possible.
For the algorithm comparison, three parameters are listed in Table 3, and the best result image should have clear lane lines with less under-detection and over-detection.
In Table 3, "Lane lines" means that percentage of lane lines is clearly detected, the higher it is, the better the result; "Under-detection" means that the percentage of lane lines is not detected, the greater it is, the more lane lines are not detected; and "Over-detection" means that the percentage of lane lines is cut into different objects, the grater it is, the more objects are on the lane lines.
Although the points found are not necessarily continuous, they are mostly concentrated on or near to the lane lines. For the new algorithm, compared with Canny or other differential operators, the algorithm does not generate too much noise and false edges, which lays a good foundation for the subsequent recursive Hough transform. The ninth row is for the corresponding recursive Hough line detection results 23,24 Table 4, the definitions are as the follows. Accuracy: It is the proportion of the sample in the test set that can be accurately detected, and it can be expressed as the follows. It can be found from Table 4 that, the results are greatly affected by the night environment lighting, resulting in the decline of detection accuracy, the analysis result shows that due to the comprehensive influence of various light sources such as neon lights and street lights in the urban night environment, all the methods caused certain misjudgments to the lane line area, resulting in a lower overall detection success rate.
In order to make the validation for the algorithm in this study, the new algorithm is compared to Li's method-Deep + V3 network. The two typical nighttime lane line images are selected as shown in Fig. 11.
In Fig. 12, the original images are from Fig. 11. It shows three step image processing procedure: lane line center line points, the recursive Hough transform and final results. Comparing to the results in Fig. 11, the new algorithm is much better than Deep + V3 network method.
At present, the data sets about lane line detection include CuLane data set, KITTI data set, TuSimple data set and Baidu Apollo lane line pixel data set 27 . KITTI data set contains less lane line feature data, the lane structure is not obvious, and the road environment irrelevant information is too much to meet the requirements of this paper. Baidu Apollo data is mainly aimed at the driving environment of automatic driving, and there is no relevant annotation requirement for the lane line structure. Therefore, in this paper, we select TuSimple data set CuLane data set as the main test and training objects.
To evaluate the new algorithm detection results, we took m0re than 400 lane line images, 200 images from TuSimple data set 43 and 200 images from CuLane data set 44 , and the algorithm mainly is compared to a semantic method (Deeplab V3 + network) 27 , and the testing and comparison results are presented in Table 5, where, in the daylight, the results between two methods are almost the same, and in the night, the new algorithm has increased the Accuracy and Recall much than the semantic method.
To show the comparison results visually, we selected two groups of sample images from the two public dataset as following. We present three images from TuSimple data set 43 and their detection results, and three  www.nature.com/scientificreports/ images from CuLane data set 44 and their lane line detection results. The testing results are described in Fig. 13 and Fig. 14 respectively. In Fig. 13, the three original images are selected from the public dataset: TuSimple data set 43 , and the three images represent three different situations, they cannot be processed by a normal image segmentation algorithm because of complex of the images. By comparing to the current semantic method 27 , the new algorithm described above can give out the good lane line detection results for the straight lane lines. The details for the comparison are presented as the follows.
In Fig. 13a, it is a two lane road image, the left lane is a continuous yellow line, and the right lane is a white line but not a continuous line in which almost 70% length have no white color; comparing to the groundtruth in Fig. 13d, the semantic method (Deeplab V3 + network) 27 creates 4 extra lane lines in addition to the two actual lanes as shown in Fig. 13g; and the new algorithm in this study can clearly detect the central lines of the two lanes as shown in Fig. 13j. The image in Fig. 13b has three lane lines, the middle lane line is short and not continuous, and the illumination in the image is uneven; the semantic method 27 can detect the four lane lines (including the shoulder line on right), but also detects an extra short line on left, as shown in Fig. 13h; and the new algorithm can give out the three lane lines clearly, see Fig. 13k. In Fig. 13c, the image has three fuzzy and intermittent lane lines and multiple vehicles, and the illumination is uneven; expect for the lane lines, the semantic method also creates an extra lane line on left side of road as shown in Fig. 13h; and the studied algorithm in this paper can extract the three lane lines exactly, as shown in Fig. 13l.
In Fig. 14, one dusk image and two night original images are chosen from the public dataset: CuLane data set 44 , the three images are from different environments, as shown in their groundtruths, they are hard to deal

Conclusion
The research content mainly includes five aspects: (1) Considering that the acquired lane line image at night, it does not necessarily have obvious lane lines, based on the length, width, interval distance of lane line (Table 1), possible vehicle speed (Table 2), road class and video acquisition frequency, a multiple video image fusion is made as the detecting image, in this way, the lane lines can be basically guaranteed showing up in each detecting image. (2) Since the influence of street lights at night on the traffic road is great, the detection region of lane lines cannot be a fixed for all the images, so a dynamic algorithm to determine the valid detection region is studied, which is an algorithm based on region merging. In the process of this study, hundreds of video images of lane lines on freeway at night are tested with 9 different algorithms, and the global thresholding, dynamic thresholding, different Canny edge detectors, Clustering analysis, fuzzy clustering analysis (FCM) and MST graph based algorithms are compared to the new method. The experimental and comparison results show that the new method (includes several algorithms) proposed in this paper can be applied for the automatic detection of the lane lines on the highway at night, and can achieve the good effect that other algorithms are difficult to obtain. Further research focuses on that the new method can automatically decide video image fusion rules for different length vehicles and different highway classes.